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ABSTRACT 

A study evaluated the higher order thinking skills 
model developed at the Mid-continent Regional Educational Laboratory. 
A primary thesis of the thinking skills model is that the majority of 
the cognitive skills considered relevant to academic success can be 
categorized ixato three distinct groups: (1) learning-to-learn skills, 
(2) content thinking skills, and (3) basic reasoning skills. These 
are further subdivided into 18 specific skills, which are generally 
presented to teachers as a single intervention, or comprehensive 
training program, which usually takes four days, spaced two to five 
weeks from preceding and subsequent training days. During the time 
between training days participating teachers collected data on the 
effectiveness of the various techniques by recording their 
perceptions of changes in student behavior, which they then used to 
make informed decisions as to which techniques worked or did not work 
well for them. These data were used for the evaluation. Subjects for 
the evaluation wei:e 77 students drawn from four sites where the 
thinking skills program was pilot tested: a large suburban school 
district, a small rural school district, a large rural district, and 
a single elementary school. For one site, the trainer used protocol 
analysis to analyze the information obtained from student interviews. 
The results indicated that all 18 of the skills involved in the 
program produced measurable effects in student behavior. (Results for 
each of the 18 skills are included.) (HTH) 
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INTRODUCTION 

The purpose of this report is to summarize the evaluation efforts 
for the higher order thinking skills model developed at the 
Mid-continent Regional Educational Laboratory. The theoretical 
framework was reported in the document entitled "A Unitary Model of 
Cognition and Instruction in Higher Order Thinking Skills" submitted to 
the National Institute of Education in 1984. 

Since that time the thinking skills model has been 
reconceptualized . Because the thinking skills strategies, which were 
evaluated were based on the reconceptualized model, we briefly outline 
that new model below. For a detailed description of that model see 
Marzano (1985) or Marzano and Hutchins (1985). 

THE THINKING SKILLS MODEL 

A primary thesis of the thinking skills model is that a majority 
of the cognital skills considered relevant to academic success can be 
categorized into three distinct groups: 1) learning-to-learn skills, 2) 
content thinking skills and 3) basic reasoning skills. We conr^ider 
each below: 

Learning-to-Learn Skills : 

The learning-to-learn skills are those which facilitate learning 
of all types. The basic assumption underlying the learning-to-learn 
skills is that learning within a classroom setting is a function of 
generalized competencies that are used in all learning situations — 
not just those related to sc^jcdrrl. If students are taught or made aware 
of these generalized competencies, they can use them in any situation 



~ school related and non-school related. There are four general 
competencies in the learning~to-lc^arn area: 1) attention control i 2) 
goal setting, 3) monitoring attitudes aud 4) self~evaluat ion. 

At tent ion control involves an awareness on the part of students o f 
wh^^n they are and are not attending to a task. Within cognitive 
psychology, two types of attention are commonly identified: automatic 
and volunt ary (Lur ia, 1983) . Automatic attention is basical ly 
reflexive in nature; voluntary attention occurs when an individual 
willfully shifts attention to or maintaii:s attention on a specific 
stimulus. The intent of the thinking skill entitled attention control 
is to provide students with an awareness of v^en they are and are not 
attending and a set of strategies for shifting attention when they so 
desire. 

Good setting is the skill of identifying an explicit outcome and 
planning activities to accomplish that outcome. Over forty years ago. 
Sears (1940) found that successful students tended to set explicit 
goals. More recently Brophy (1982) found that successful students 
tended to set goals that were challenging but reasonable. Bandura and 
Schunk (1981) found that classroom goals should be short term 
(proximal) rather than long term (distal) to be most effective. 

Once a goal is set an individual will generally consider some high 
level controlling attitudes relative to the goal. A high level 
controlling attitude can be described as a basic operating principle 
which governs behaviour. These principles are so important to human 
behaviour that some theorist have hypothesized the existence of a 
specific type of memory housing then called "executive memory." 
(e.g. Sternberg, 1984). Within the thinking skills program students 



are taught to identify and foster specific attitudes which facilitate 
the accomplishment of academic goals (e.g. commitment to precision and 
involvement) • 

During the attainment of a goal , effective learners commonly 
engage in self-evaluation techniques in which they identify what is 
working and what is not working relative to the goal. This can be done 
both formatively and summatively. That is, students can monitor the 
effectiveness of their activities while they engage in a task and after 
they have completed the task. 

Content Thinking Skills 

The purpose of the content thinking skills is to provide students 
with a set of information processing strategies which facilitate the 
learning of academic content sometimes referred to as domain specific 
knowledge (Doyle, 1983). There are four content thinking skills within 
the thinking skills program: 1) concept attainment, 2) pattern 
recognition, 3) synthesizing and 4) proceduralization. 

Concept attainment is the process of attaining the culturally 
accepted labels for a set of experiences. According to Klausmeier and 
Sipple (1980) a concept is the "socially accepted meaning of one or 
more words which express the concept." For example, the word dog is 
the label society uses to represent the conceptualization of a set of 
four-legged animals with certain characteristics. Within the thinking 
skills progran concept attainment refers to, 1) the development of the 
experiential base for the concept, 2) the recognition of the auditory 
label for the concept, 3) the recognition of the orthographic label for 
the concept and 4) the development of an accurate description of the 
concept . 
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Patterns are organizational structures v*iich hold together large 
blocks of information. There are two levels of patterns taught within 
the thinking skills program - basic patterns and macro-patterns. Basic 
patterns are used to organize linguistic information from the size of a 
paragraph to a few pages, or even a chapter of a textbook. 
Macro-patterns are used to organize even larger blocks of linguistic 
information (e.g. an entire book). 

Synthesis is the process of expressing information read or heard 
into a streamlined version referred to as a macro-structure (Rintsch 
and van Dijk, 1978). This generally involves such techniques as 
identifying summary statements, generating summary statements when none 
exist and subsuming specific concepts under more general concepts. 

Proceduralization is the process of 1) identifying the relevant 
steps in a specific process, 2) rehearsing these steps with the intent 
of "smoothing them out" (dropping steps that are inefficient and adding 
steps that are more efficient), 3) practicing the process until it 
reaches a level of automaticity. This sequence for procedure learning 
was first described by Fitts (1964) and later expanded by Anderson 
(1982). 

Basic Reasoning Skills 

Reasoning skills are those considered basic to many cognitive 
tasks. According to Anderson (1983) reasoning skills can be subdivided 
into three general classes: 1) those that foster storage and retrieval 
of information, 2) those that match one set of information with the 
other and 3) those that drastically restructure old information or 
build new information. 



1. Storage and Retrieva l 
There are two basic storage and retrieval skills: 1) deep 

processing and 2) Toemory frameworks. Deep processing is the use of 

imagery to enhance incoming informat ^.on • According to Sheikh (1983) » 

imagery involves the creation of mental pictures, sounds , tastes , 

smells, noises and emotions about information. Apparently, with 

practice, people can enhance their imagery skills (Bellezza, 1983). 

Memory frameworks create "slots" in which sequential information 

can be deposited using deep processing. Within this category are loci 

methods (Ross & Lawrence, 1968) and pegword methods (Lindsay & Norman, 

1977). 

2. Matching Skills 

Matching skills are those which enable an individual to identify 
how incoming information is similar to and different from information 
stored in long term memory. There are five types of matching skills, 
1) categorization, 2) e^^trapolat ion, 3) analogical reasoning, 4) 
evaluation of logic and 5) evaluation of value. 

According to Mervis (1980), categorization is an essential skill 
because "by categorizing a person is able to render the unfamiliar 
familiar and because one is able to generalize about an object based on 
knowledge about its category, one is able to know more about the object 
than just i*iat can be ascertained by looking at it" (p 279). 

Within the thinking skills program, categorization is practiced as 
an adjunct skill to concept development. That is, students are asked 
to categorize new vocabulary words by their relevant attributes. 
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Extrapolation is the process of oatching the pattern of 
information read or heard with that of information from a different 
subject area or a different context. According to Alston (1964) this 
type of thinking is similar to the basics of metaphor. 

Analogical reasoning is one of the most commonly included elements 
within thinking skills instructional models. According to Alexander 
(1984) few intellectual skills are as pervasive or as essential as ti>e 
ability to reason analogically. Within the present program, analogical 
thinking is taught using an adaptation of the four step process 
('^escribed by Sternberg (1977). 

Evaluation of logic is the process of matching the structure of 
information with some formalized system of logic. Within this model, 
the logic system developed by Toulmin (1958) (Toulmin, Rieke & Janik, 
1977) is used as the logic criterion. 

Evaluation of value is the process of matching information to 
some internalized system and then analyzing the logic of that value 
system. Spiro (1980) has stated that this "attitudinal" characteristic 
of thinking is the central aspect of cognition because it allows one to 
see the information base from which judgments are made* 

3. Knowledge Building & Reorganization 
There are three knowledge building and restructuring skills within 
the program: 1) elaboration, 2) problem solving and 3) invention. 

Elaboration refers to inferring information not explicitly stated, 
various categories of inference have been proposed by teachers and 
theorists (e.g. Bruce & Schmidt, 1974; Warren, 1979). Within this model 
three types of inferences are used: 1) elaboration of characteristics, 
2) elaboration of casuality, 3) elaboration of general background and 



and 4) elaboration of author purpose. 

Problem-solving occurs i*ien an individual must "fill-in" missing 
information. This is at the core of all problems; a goal is desired 
and infonuation necessary to accomplish the goal is missing. Within 
this model problem solving algorithms have been developed for general, 
everyday problems , mathemat ic al probletris , science problems , problems of 
constraining conditions and problems of unusual contexts. 

Invention is the process of creating new information or new 
products. This involves conceiving cr Ideal, planning for its 
development, creating a working model and then revising or fine tuning 
the model until it reaches a polished state. Within mainstream 
education this process is generally taught as composing in written 
form. Within the thinking skills program it is taught as a general 
heuristic for the creation of any product* 

THE INTERVENTION 

The eighteen skill areas described above are generally presented 
to teachers as a single intervention - a comprehensive program. The 
order of presentation to teachers follows that described above. Th&t 
is, teachers are first presented techniques for the learnlng-to-learn 
skills, then techniques for the content thinking skills and finally 
techniques for the basic reasoning skills. The training usually takes 
four days (1 day for the learning-to-learn skills, 1 day for the 
content thinking skills, and 2 days for the reasoning skills). Each 
day of the training is spaced 2 to 5 weeks from preceding and 
subsequent training days. During the time between trainings 
participating teachers collect data on the effectiveness of the various 
techniques. At the end of the four days of training participating 
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teachers use the data to make informed decisions as to v*iich techniques 
work well for them and which techniques do not workwell. For the most 
part it was the teacher collected data that was used for the 
evaluation. 



DEPENDENT VARIABLES 

Although the eighteen thinking skills are presented as a single 
intervention, they are hypothesized to affect different student 
behaviours. That is, the eighteen different components of the 
independent variable (the thinking skills program) were hypothesized to 
affect different depeadent variables. 

Below the dependent measures for each of the eighteen areas are 
briefly described. 





Skill Area 


Hypothesized 
Effect 


Dependent 
Measure 


1. 


Attention 
control 


-increased task 
engagement 


-engaged time 
-engagement rate 
-teacher observation 


2. 


Goal setting 


-incre*. sed success 
in academic goals 


-success rate 
-teacher observation 


3. 


Monitoring 
att itudes 


-increased effort 
attribution relative 
to goals 


-student interviews 
-teacher observation 


k. 


Self-evaluation 


-increased metacognitive 
awareness of effect ive 
and ineffective actions 


-student interviews 
-teacher observation 


5. 


Concept 
attainment 


-understanding of basic 
meaning of target 
concepts ; recognition 
of phonemic and 
orthograph represent- 
ation of the concept 


-teacher made 

vocabulary and 

spelling tests 
-teacher observation 


6. 


Pattern 
recognition 


-increased ability 
to organize large 
blocks of information 


-teacher made tests 
-teacher observation 


1. 


Synthesizing 


-Increased ability 
to express the macro* 
structure of informa- 
tion 


'-ti'.acher observation 
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8. 


Proceduralizing 


-increased ability 
to internalize complex 
procedures 


-teacher 


observation 




beep processing 


-Increased ability co 

elaborate on the 

imagary characteristics 

of information 
-increased ability to 

recall information 


-teacher 


oftservsrlca 


10. 


Memory framework 


-increased ability to 
organize information 
into sequential sets 

-increased ab ili ty to 
recall information 
sequentially 


-teacher 


observation 


11. 


Categorising 


-increased ability to 
organize concept s into 
semantic categori es 

-increased ability to 
identify shared 
attribut es among sets 
of concepts 


-teacher 
-teacher 


made tests 
observat ion 


12. 


Exploration 


-increased abil i ty to 
recognize similarities 
and differences 
between sets of 
informat ion from 
di f ferent sources 


-teacher 
-teacher 


made tests 
observation 


13. 


Analogical 
reasoning 


-increased ability to 
solve analogy problems 


-teacher 
-teacher 


made tests 
observation 


14. 


Evaluation of 
logic 


-increased ability 
to determine v^ether 
a claim is supported 
or unsupported 


-teacher 
-teacher 


made tests 
observat ion 


15. 


Evaluation of 
value 


-increased ability to 
identify the assumptions 
underlying personal 
values and the basis for 
those assumptions 


-student 
-teacher 


interviews 
observ lotion 


16. 


Elaboration 


-increased ability to 
identify unstated 
characteristics , causes 
and intentions 


-teacher 


observat ion 


17. 


Problem solving 


-increased metacognitive 
2:varene:js of processes 
for solving specific 
problems 

-increased ability to 
solve specific problem 
types 


-teacher 
-student 


observat ion 
interviews 




Invention 


-increased ability to 


-teacher 


observation 



conceived a product and 
develop it to a final 
format 
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SUBJECTS 

The subjects for the evaluation were drawn from four sites where 
the thinking skills program was pilot tested. Those sites are 
described below: 

Site #1 was a large suburban school district serving over 6000 students 
of mixed socio-economic strata. Twelve teachers from this district 
participated in the I'lidy, They represented the following grade 
levels: 

Grade Level No of Teachers 
Primary 4 



Upper Elementary 

Junior High 

Senior High 



Site #2 was a small rural school district serving about 700 students. 

Twenty-four teachers from the district participated in the study. The 

grade levels represented are listed below: 

Grade Level No>of Teachers 

Primary 8 

Upper Elementary 9 

Junior High 4 



Senior High 



Site #3 was a large rural district serving about 800 students, 

Thirty-five teachers participated in the study: 

Grade Level No. of Teachers 

Primary 



Upper Elementary 16 

Junior High 4 



Senior High 10 



Site #4 was a single elementary school. Here six teachers participated 
in the study: 

Grade Level No. of Teachers 

Primary 2 

Upper Elementary 4 



Including all four sites, the evaluation study consisted of the 
following distribution of teachers: 

Grade Level No. of Teachers 

Primary 



Upper Elementary 
Junior High 



Senior High 



4 


+ 


8 


+ 


5 + 2 =19 


3 


+ 


9 




16 + 4=32 


i 


+ 


4 


+ 


4 . =10 


3 


+ 


3 


+ 


10 =16 



=17 

The 77 teachers involved in the study pilot tested the thinking 
skills techniques on over 1900 students. 
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DATA COLLECTION 



For sites 1-3 data v;ere collected by participating teachers 
between training sessions. That is^ after training session #1 
participating teachers collected data on the techniques presented 
during sessional; after session #2 teachers collected data on the 
techniques presented in session #2; etc. At site #4 the data were 
collected by the trainer. Site #4 utilized a different format. 
Participating teachers first went through all four days of the 
training. The techniques were then pilot tested in an after school 
program in which students voluntarily participated. The trainer was 
present at all sessions and utilized those sessions to collect types oi 
data that could not be collected from participating teachers. 



All tests used to assess academic performance were teacher made. 
No reliability or validity data were collected on those instruments. 
Similarly, the teacher observation data were collected in anecdotal 
fashion. That is, for the observational data, teachers simply recordec 
their perception of student charges in behaviour. It was only for the 
success rate and engagement rate data collected by the trainer that 
formal observation systems were utilized. The trainer also used a 
formal sys tern of protocal analys is to analyse the infonnat ion obt ai ned 
from student interviews. 

LIMITATIONS 

The limitations of the present evaluation effort were many. One 
obvious limitation was that the effects of the varicvis thinking skill 
techniques can not be considered independent since the training was 
presented in a sequential fashion. Even though the theory base 
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underlying the various skill areas would indicate that their affects 
should be relatively independent, the evaluation design was such that 
the skills taught earlier in the sequence could mask the effects of 
skills taught later in the sequence or create an effect not related to 
the technique under investigation. 

For most of the skill areas the evaluation design was \^at 
Campbell and Stanley (1963) refer to as a one-shot case study. In such 
designs the dependent variable is measured after the intervention. 
According to Campbell & Stanley (1963) such a design has the 
followingthreats to internal and C-:ternal validity: 

-history 
-maturation 

-interaction of selection and treatment 
For those skill areas for which pretest data could be collected, 
a prete8t-pos ttest design was used. According to Campbell and Stanley, 
the design has the following threats to internal and external 
validity: 

-history 

-maturation 

-testing 

-ins trument at ion 

-interaction of selecLion,maturation, etc. 
-interact ion of testing and treatment 
-interaction of selection and treatment 



RESULTS 



Below the results for each of the 18 skill areas are discussed. 
Since all teachers in the study did not collect data on all skill 
areas, the number of teachers reporting is included in each 
discussion: 
1 . Attention Control 

The effect for attention control was measured using teacher 
anecdotal comme:its, engaged ti.Qe and engagement rate. The teacher 
anecdotal comments were reported by 65 teachers. Of those 51 reported 
that students were more aware of when they were and were not engaged 
and exhibited more control over their attending behavior. Fourteen 
teachers reported no noticeable change in student attending 
behaviours . 

Engaged time and engagement rate were collected by the 
investigator in a pre-post fashion in 6 classrooms. Engaged time is 
expressed as a proportion of total class time. Engagement rate 
is expressed as the ratio of time engaged over the time devoted to 
academic tasks. The results for engaged time and engagement rate are 
reported below: 



Engaged Time 



Pre-test 
Mean 



Post-test 
Mean 



Difference 




T-value 



Sig 



.75 



.73 



.02 



.13 
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ENGAGEMENT RATE 
Pre- test Post- test 

Mean Mean Difference df T-value Sig 

,81 ,91 ,10 5 3,01 ,03 

T-test results showed no significant difference in engaged time 
but did show a significant difference in engagraent rate. This 
indicates that students did not experience iQore engaged time probably 
due to the increased amount of non-instructional time necessary to 
administer the technique. However, students did exhibit a significant 
increase in engagment rate while instruction was occurring, 
2, Goal Setting 

Goal setting w&s evaluated using teacher anecdotal data and 
success rate data. Anecdotal data were collected by 61 teachers. Of 
those ,53 reported that having students set explicit short term goals 
increased student success as measured by responses to teacher 
questions and in-class assignments. Fifteen participating teachers^ 
reported pre- and post-test classroom exercise data. Scores were 
transformed to proportions and two-tailed t-tests for dependent groups 
run: 

Pr e- 1 e s t Pos t-test 

Mean Mean Difference df T-value Sig 

,81 ^88 ^07 175 4,05 ,001 

Pre-post engagement rate data wo.re collected on 6 teachers. 

The results for a two-tailed, t-test for dependent groups are: 

Pre-test Pos t-test 

Mean Mean Difference df T-value Sig 

,79 ,91 ,12 5 8,17 .001 
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Both pre-CesC and posC-cesC analyses showed significant gains at 
the .01 level. 

3. Monitoring Attitudes 

Student att itude change was assessed using teach ing observat ion 
and student interview data. Teacher observation data were reported by 
36 teachers. Of those, only 7 reported an observable change in student 
behavior as a result of the attitude monitoring techniques. However, 
29 teachers also reported that they experienced an increased ability to 
communicate with the more troublesome students about specific problems 
within the class* 

Self report data were collected from 7 students. Of those, 5 
reported that they could see why certain attitudes abont certain 
classes were "getting in their way." 

4. Self-evaluation 

The effect of the self-evaluation technique were assessed using 
teacher observation and student interview data. Anecdotal data were 
reported by 34 of the 77 participating teachers. Of those, 29 stated 
that the self-evaluation techniques began to foster a self-awareness on 
the part of students of efficient and inefficient act ivities relat ive 
to completing a task. The remaining five reported no noticeable effect 
in students. 

Interview data were collected from 7 students - all 7 of v^ich 
reported that the self-evaluation techniques made them more aware of 
when the vere and were not being efficient. 
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5. Concept Attainment 

Teacher observation and teacher made test data were collected to 
measure the effects of the concept attainment strategies. Sixty-five 
participating teachers reported on the concept attainment process. Of 
those, 54 stated that they saw a definite improvement in students' 
knowledge of concepts as a result of the strategies. 

Seventeen teachers reported pre-post scores on teacher made 
vocabulary tests. Results from a two-tailed, t-test for dependent 
groups are reported below: 

Pre-test Post- test 

Mean Mean Difference df T-value Sig 

.83 ^94 Al 145 10.17 .001 

Eleven teachers reported pre-post scores on teacher made spelling 

tests. Results from a two-tailed t-test for dependent groups were: 

Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.74 .81 ^08 97 11.12 .001 

Both the vocabulary and spelling tests showed significant 
increases in scores as a r^isult of the concept development techniques. 

6. Pattern Recognition 

Data on pattern recognition were collected via teacher observation 
and teacher made tests. Sixty-one teachers provided anecdotal 
data. Fifty-four of those, indicated that the pattern recognition 
techniques significantly improved students' abilities to understand 
material they read. 
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Twelve teachers reported pre-post test data on student 
comprehension. Results from a two-tailed t-test for dependent groups 
are reported below: 

Pre-test Post-test 

Mean Mean Difference df T-value Sig 

•^^ -83 .09 168 15.51 .QQl 

Results indicated a significant increase in students' comprehension 

ability as a result of the pattern recognition training. 

7. Synthesizing 

Teacher observation was the only type of data collected on the 
skill of synthesizing. Twenty-three teachers reported on this 
technique. Of those. 17 indicated significant increase in study 
ability to synthesize info^ma^-:on as a result of the technique. 

8. Procedural izing 

Due to its relatively late inclusion in the thinking skills model, 
only four teachers reported on the effects of the proceduralizing 
techniques. All four indicated that this was a very difficult skill to 
teach students. None saw any immediate change in student behavior but 
all felt that over time this skill would effect useful changes in 
student behavior. 

9. Deep Processing 

Evaluation data for the technique, deep processit^. vrere collected 
via teacher observation. Fifty-nine teachers reported. Forty-seven of 
those indicated that the deep processing technique increased students' 
abilities to elaborate on the imagary characteristics of information 
and to recall information once it had been deep processed. 
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10. Memory Framework 

Teacher made tests were used to evaluate the use of memory 
frameworks. Twenty-five teachers reported pre-post results: 
Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.46 ^^8 .J2 161 25.71 .001 

A significant increase in students' abilities to recall 

information sequentially is indicated frora these findings. 

11. Categorizing 

Teacher observation and teacher made test data were used to assess 
the categorizing technique. Seventeen teachers provided anecdotal 
comments. Fourteen indicated that categorizing helped students 
understand the meaning of concepts and their defining attributes 

Six teachers turned in pre-post test data. The results of a 
two-tailed , t-test for dependent groups were: 
Pre-test Post-test 

Mean Mean Difference df T-value Sig 

,85 .88 ,03 4^ 1.04 N.S. 

Although there was a slight gain in student performance on tests 

of concept knowledge, this gain was not significant statistically, 

12. Extrapolation 

Extrapolation was assessed using teacher observation data and 
teacher made tests. Observation data were supplied by 21 teachers. 
Seventeen reported that the technique increased students' abilities to 
see relationships between seeningly unrelated sets of information. 

Ten teachers reported test data. The results of two-tailed, 
t-test for dependent groups are reported below: 
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Pre-test Post-test 

Mean Mean Difference df T-'value Sig 

,81 ^95 .JA 65 5.01 >001 

The analyses indicated a significant increase in students' 

abilities to extrapolate information. 

13. Analogical Reasoning 

Analogic al reasoning was assessed using teacher observat ion and 
teacher made tests. Thirty-seven teachers reported anecdotal data. Of 
those, 26 stated that direct teaching of the technique for analogical 
reasoning improved students' abilities to solve analogy problems 

Ten teachers reported pre-post test data. A two-tailed t-test for 
dependent groups showed significant results at the .001 level 
Pre-test Post-test 

Mean Mean Difference df T-value Sig 

.86 ^94 ^08 91 7.35 ,001 

14 . Evaluation of Logic 

Both teacher observation and teacher made test data were used to 
assess the skill, evaluation of logic. Twenty-two teachers reported 
anecdotal data. Seventeen stated that they observed an increase in 
students' abilities to evaluate the logic of a claim. However, a 
two-tailed t-test for dependent groups on the test data reporte'I by 
three teachers did not show significant results 
Pre-test Post- test 

Mean Mean Difference df T-value Sig 

.79 .81 .03 19 .79 N.S 
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15. E valuation of Value 

Evaluation of value was assessed usiiig teacher observation and 
student self-report data. Twenty five teachers reported anecdotal 
data. Of thosfij 21 stated that the evaluation of value process 
increased students' abilities to identify the assumptions underlying 
their assignment of value and to identify different points of view 
which might generate different value judgments. 

Seven students were interviewed about the effects of the 
evaluation of value process. All seven students indicated that the 
process provided them with a new level of insight into v*iat attitudes 
and values are and how they are formed. 

16 . Elaborat ion 

Data on elaboration were collected via teacher observation only. 
Forty-one teachers reported . Of these , 37 stated that the elaborat ion 
process increased students' abilities to identify unstated 
characteristics^ causes and intentions. That is, the elaboration 
process provided a framework \^ich allowed students To differentiate 
literal from implied information and to "fill in" implied information. 

17. Problem Solving 

Problem solving was assessed using teacher observation and stude t 
interviews. Eighteen teachers provided observational data. Twelve 
indicated that the problem solving techniques improved students' 
abilities to solve specific content related problems. Self-report data 
were collected from five students, all of \^om indicated that the 
problem solving techniques presented them with a systematic way to 
approach and solve problems. 
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18 • Invention 

Invention was assessed via teacher observation only. Twelve 
teachers reported. Of those, 9 stated that the invention process 
significantly improved students' abilities to conceive of, develop and 
carry through to completion projects involving the creation of new 
products. 

DISCUSSION 

All eighteen of the skills involved in the thinking skills program 
produced measurable effects in student behavior to one degree or 
another. This is not surprising in light of the fact that the thinking 
skills modeled discussed in this report is . actually a fourth generation 
model. That is, three other versions of the program were informally 
field tested over a period of two years. Basci on these field 
testings, components of the program were added,, deleted and modified, 
to produce the present model. It would be expected, then, that those 
skills remaining after such a filtering process would produce positive 
effects. 
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